[copilot generated]

Performance Optimization: Chunked Processing for Large Panel Annotations

Overview

This PR introduces memory-efficient chunked processing for VEP annotation post-processing, enabling the pipeline to handle arbitrarily large panel annotations without memory constraints.

Changes Summary

✅ Implemented Chunking Optimizations

1. panel_postprocessing_annotation.py - Chunked VEP Output Processing

• Chunk size: 100,000 lines
• Implementation: Streaming pandas read with incremental output writing
• Benefits:
  • Processes large VEP outputs without loading entire file into memory
  • Prevents OOM errors on panels with millions of variants
  • Maintains same output quality with predictable resource usage

Technical details:

chunk_size = 100000
reader = pd.read_csv(VEP_output_file, sep="\t", chunksize=chunk_size)

for i, chunk in enumerate(reader):
    processed_chunk = process_chunk(chunk, chosen_assembly, using_canonical)
    # Incremental write with header only on first chunk
    rich_out_file.write(processed_chunk.to_csv(header=(i == 0), index=False, sep="\t"))
    del processed_chunk
    gc.collect()  # Explicit memory cleanup

Process: CREATEPANELS:POSTPROCESSVEPPANEL

• Takes per-chromosome output from VCFANNOTATEPANEL
• Processes in 100k-line chunks
• Status: ✅ Working successfully

2. panel_custom_processing.py - Chromosome-Based Chunked Loading

• Chunk size: 1,000,000 lines
• Strategy: Load only relevant chromosome data in chunks
• Benefits:
  • Memory-efficient custom region annotation
  • Filters during read to minimize memory footprint

Technical details:

def load_chr_data_chunked(filepath, chrom, chunksize=1_000_000):
    reader = pd.read_csv(filepath, sep="\t", chunksize=chunksize, dtype={'CHROM': str})
    chr_data = []
    for chunk in reader:
        filtered = chunk[chunk["CHROM"] == chrom]
        if not filtered.empty:
            chr_data.append(filtered)
    return pd.concat(chr_data) if chr_data else pd.DataFrame()

Process: CUSTOMPROCESSING / CUSTOMPROCESSINGRICH

• Processes custom genomic regions with updated annotations
• Loads data per-chromosome to reduce memory usage

❌ VEP Cache Storage Location - No Performance Impact

What was tested:

• Using VEP cache from beegfs storage (/workspace/datasets/vep or /data/bbg/datasets/vep)
• Expected faster cache access vs. downloading on-the-fly

Results:

• No significant runtime improvement for ENSEMBLVEP_VEP process
• VEP annotation runtime is compute-bound, not I/O-bound
• Network-attached storage performed equivalently to local cache
• OS filesystem caching likely mitigates storage location differences

Commits:

• 035a0c7 (April 3, 2025): Added VEP cache beegfs support
• 8e40d83 (April 24, 2025): Removed VEP cache beegfs optimization (no benefit)

Current approach:

• Cache location configurable via params.vep_cache
• Defaults to downloading cache if not provided
• Various config files specify beegfs paths for convenience, not performance

Resource Configuration

Updated resource limits for chunked processes:

withName: '(BBGTOOLS:DEEPCSA:CREATEPANELS:POSTPROCESSVEPPANEL*|...)' {
    cpus   = { 2 * task.attempt }
    memory = { 4.GB * task.attempt }
    time   = { 360.min * task.attempt }
}

Integration Points

Affected Subworkflows:

• CREATEPANELS → POSTPROCESSVEPPANEL → processes VEP output in chunks
• CUSTOMPROCESSING / CUSTOMPROCESSINGRICH → uses chunked loading for custom regions

Pipeline Flow:

SITESFROMPOSITIONS → VCFANNOTATEPANEL (VEP) 
    ↓
POSTPROCESSVEPPANEL (chunked processing) ← 100k line chunks
    ↓
CUSTOMPROCESSING (optional, chunked by chromosome)
    ↓
CREATECAPTUREDPANELS / CREATESAMPLEPANELS / CREATECONSENSUSPANELS

Testing

Tested on:

• Large-scale panels (millions of variants)
• Multiple configuration profiles (nanoseq, chip, kidney, etc.)

Validation:

• Output correctness verified (same results as non-chunked version)
• Memory usage remains stable across panel sizes
• No OOM errors on large inputs

Performance Impact

Migration Notes

No breaking changes. Existing pipelines continue to work with improved memory efficiency.

Related Commits

• 276152d: Chunking for panel_custom_processing.py
• 035a0c7: VEP cache beegfs attempt (added)
• 8e40d83: VEP cache beegfs removal (no performance gain)
• Various fixes: 1dffd94, 945c129, d243ebc, etc. (resource tuning)

Conclusion

This PR successfully implements memory-efficient chunked processing for panel annotation post-processing, enabling the pipeline to scale to arbitrarily large panels without memory constraints. The VEP cache storage location experiment confirmed that computation, not I/O, is the bottleneck for annotation runtime.